Apparatus for synthesizing three-dimensional images to visualize surroundings of vehicle and method thereof

ABSTRACT

The present invention relates to three-dimensional visualization of the surrounding images of a vehicle, and comprises the steps of: enabling a plurality of wide angle cameras provided at a vehicle to receive a plurality of photographed images for reference patterns formed on the ground; extracting feature points from the photographed reference patterns and estimating a relative location and an installation angle of each camera using the known physical location information of the extracted feature points; obtaining optical parameters comprising an optical center of a lens for correcting lens distortion using the photographed images and mapping each image on a surface of a Gaussian sphere using the obtained optical center; changing an angle and distance such that the Gaussian sphere and the real reference patterns correspond to each other using the relative location and the installation angle of the estimated camera and arranging the images in a three-dimensional virtual space; and obtaining a three-dimensional single image by mapping each image arranged in the three-dimensional virtual space to an inner surface of the three-dimensional sphere corresponding to one large sphere.

TECHNICAL FIELD

The present invention relates to a multichannel image-based driverassistance system and, more particularly, to an apparatus and method forsynthesizing 3D images for the purpose of visualizing the surroundingsof a vehicle, which visualize images of the surroundings of a vehiclephotographed using a plurality of cameras in the form of athree-dimensional (3D) image, thereby being able to enhance reality.

BACKGROUND ART

As a method of visualizing the surroundings of a vehicle that belongs tocurrently known apparatuses, there is a top view system based onmultichannel image synthesis. This is used to synthesize images of thesurroundings of a vehicle that represent blind zones that are presentout of the field of view of a driver. The top view output apparatusoutputs a plurality of images obtained via cameras installed in frontof, in back of, on the left of and on the right of a vehicle in the formof a single continuous image. That is, it generates a 2D plane image inthe form of a top view that seems to be viewed from the upper end of thevehicle.

When a top view output apparatus using a plurality of cameras is appliedto a bird's eye image, attention should be paid to installation so thatthe cameras basically use fisheye lenses having a field of view of about180 degrees and thus common areas are ensured between neighboringcameras.

Conventional top view output apparatuses model ground plane informationaround a vehicle by combining algorithms, including lens distortioncorrection, pattern extraction, and the conversion of a point of viewvia homography, after images have been obtained from cameras in atolerance correction process. These top view output apparatuses aresystems that intuitively visualize information around a vehicle andcontinuously represent ground plane information, such as parking lines,around the vehicle over an omnidirectional range of 360 degrees aroundthe vehicle.

However, the top view output apparatuses have several realisticlimitations. First, the top view output apparatuses cannot guarantee thecontinuity of a ground object, other than a ground plane, because theydo not take into consideration the continuity of a ground plane object.Furthermore, when images are synthesized, the distortion of a groundobject is considerably increased because the top view output apparatusconsiders the ground object to be a ground plane object. As a result, asynthetic image displayed by the top view output apparatus appears to bea planar image that is somewhat different from a real image of thesurroundings of a vehicle. Accordingly, the top view output apparatuseshave low visibility with respect to a ground object.

Second, most of the currently developed top view output apparatusesrepresent a region of interest with a specific boundary set around avehicle. The currently developed top view output apparatuses generallyrepresent only an adjacent area, narrower than an area whose boundary is2 m away from the front, back, and left and right sides of the vehicle,around a vehicle. If they represent an area wider than an area whoseboundary is more than 2 m away from a vehicle, the degradation of imagequality is serious and distortion is high, and thus it is difficult toapply them to practical use. In particular, the problem of thedegradation of image quality occurs because information about a remotearea in input images is significantly insufficient compared to that of aremote plane image whose representation is desired. Furthermore, if uponreverse parking, it is considered that a series of parking steps thatare performed by a driver include the determination of the stopped andmoving states of objects around a vehicle and the relative distances tothe objects, the visual area actually supported by the top view outputapparatuses may be viewed as too narrow to detect/determine theabove-described elements. Accordingly, the top view output apparatuseshave narrow visual areas.

Third, the distortion of a ground object, other than a plane, can beeasily found in a synthetic image because the top view outputapparatuses make information about the surroundings of a vehicle planar.This phenomenon caused by distortion in an algorithm is indispensable interms of the characteristics of its principle. In particular, thephenomenon of the distortion of a ground object may become a criticalproblem when a driver interprets a synthetic image displayed via such anoutput apparatus. For example, if the synthesis device outputs adistorted image of an infant near a vehicle and a driver does notrecognize an infant and drives the vehicle without taking any measures,fatal consequences may result. Accordingly, the top view outputapparatuses perform impractical visualization over a wide range.

DISCLOSURE Technical Problem

An object of the present invention is to provide, among vehicleassistance systems, an apparatus and method for synthesizing 3D imagesfor the purpose of visualizing the surroundings of a vehicle, whichcompensate for the tolerance of cameras and synthesize a 3D image, inorder to visualize images of the surroundings of a vehicle photographedusing a plurality of cameras in the form of a 3D image.

Another object of the present invention is to provide an apparatus andmethod for synthesizing 3D images for the purpose of visualizing thesurroundings of a vehicle, which synthesize all objects in ground andnon-ground planes photographed using a plurality of cameras, therebyimplementing a visualized image of information about the surroundings ofa vehicle so that the visualized image has reality that is similar tothat of a real one.

The objects that the present invention is intended to achieve are notlimited to the above-described objects.

Technical Solution

In order to accomplish the above objects, the present invention providesan apparatus for synthesizing three-dimensional (3D) images, includingan image input/output unit configured to receive photographed images ofa reference pattern formed on a ground and a non-ground plane from aplurality of wide angle cameras mounted on a vehicle, and to output thephotographed images; an image arrangement estimation unit configured toextract feature points from the reference patterns of the image inputfrom the image input/output unit, and to estimate a relative locationand installation angle of each of the cameras using known physicallocation information of the extracted feature points; a sphere mappingunit configured to obtain optical parameters for the correction of lensdistortion including an optical center of a lens using the photographedimages, and to map each of the images to a surface of a Gaussian sphereusing obtained optical center; a virtual space arrangement unitconfigured to, using the relative location and installation angle of thecamera estimated by the image arrangement estimation unit, change anangle and distance of the Gaussian sphere so that the Gaussian spherecoincides with the real reference patterns, and arrange the Gaussianspheres in a 3D virtual space; and a single image acquisition unitconfigured to obtain a single 3D image by mapping the images arranged inthe 3D virtual space onto an inner surface of a 3D sphere correspondingto a single large sphere.

The image arrangement estimation unit may set information about theestimated relative location and installation angle of the camera.

The virtual space arrangement unit may use an optimization algorithm forestimating change parameters when changing the angle of rotation anddistance of movement of the Gaussian sphere. The optimization algorithmmay be particle swarm optimization (PSO), gradient descent (GD), orleast mean square estimation (LMSE).

In order to map an image corresponding to the central portion of each ofthe images, the single image acquisition unit may divide the surface ofthe 3D sphere by the number of cameras, and may map only an image fromthe most adjacent camera.

The single image acquisition unit may divide the inner surface of the 3Dsphere for individual longitudes based on the number of cameras, maydivide one surface of the 3D sphere divided for the individuallongitudes into ground and non-ground planes, and may perform mappingfor individual latitudes.

In order to accomplish the above objects, the present invention providesa method of synthesizing 3D images, including receiving, by a pluralityof wide angle cameras mounted on a vehicle, a plurality of images ofreference patterns formed on a ground; extracting feature points fromthe photographed reference patterns, and estimating a relative locationand installation angle of each of the cameras using known physicallocation information of the extracted feature points; obtaining opticalparameters for the correction of lens distortion including an opticalcenter of a lens using the photographed images, and mapping each of theimages onto a surface of a Gaussian sphere using the obtained opticalcenter; using the estimated relative location and installation angle ofthe camera, changing an angle and distance of the Gaussian sphere sothat the Gaussian sphere coincides with the real reference patterns, andarranging the Gaussian spheres in a 3D virtual space; and obtaining asingle 3D image by mapping the images arranged in the 3D virtual spaceonto an inner surface of a 3D sphere corresponding to a single largesphere.

Arranging the Gaussian spheres in the 3D virtual space may be performedby using an optimization algorithm, such as PSO, GD, and LMSE, in orderto estimate change parameters when changing the angle of rotation anddistance of movement of the Gaussian sphere.

Obtaining the single 3D image may include mapping the images arranged inthe 3D virtual space onto areas of the inner surface of the 3D spheremost adjacent to a center of the 3D sphere.

Obtaining the single 3D image may include dividing the inner surface ofthe 3D sphere for individual longitudes based on a number of cameras inorder to map images corresponding to central portions of the images;dividing the inner surface of the 3D sphere divided for the individuallongitudes into ground and non-ground planes based on a latitudecorresponding to the ground plane; and dividing the images arranged inthe 3D virtual space into ground and non-ground planes, and performingmapping onto corresponding areas of the inner surface of the 3D sphere.

Advantageous Effects

As described above, the present invention is directed to a method andapparatus for synthesizing 3D images, which visualize information aboutthe surroundings of a vehicle in the form of a 3D image. The presentinvention can enhance the visibility of objects around a vehicle becauseall objects located in ground and non-ground planes around the vehicleare represented in the form of a 3D image, and can improve the accuracyof the recognition of the objects around the vehicle because the objectsare presented using a 3D image having low distortion.

Furthermore, the present invention can overcome a limitation in whichthe performance of homography used in the conventional synthesis methodis unstable depending on the accuracy of the selection of patterns usedin correction and replace a series of steps performed by homography onlyby adjusting three-axis rotation variables (phi, theta, psi), therebyensuring the stability of operation of the system.

When an operator attempts to synthesize an image related to a vehicleunder manufacture before factory shipment or a vehicle under repairusing the apparatus for synthesizing 3D images according to the presentinvention, a desired image can be synthesized only by adjustingthree-axis variables. Furthermore, even when an operator attempts acorrection task in a space in which there are no reference patterns thatsatisfy predetermined conditions, a synthetic image can be correctedonly by adjusting three-axis variables, and thus the present inventionis advantageous in that operational convenience and scalability isexcellent.

Moreover, a 3D space around a vehicle is modeled, thereby providingvarious angles of field for blind-zones that have been problematic.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating wide angle cameras and an apparatus forsynthesizing 3D images according to an embodiment of the presentinvention;

FIG. 2 is a diagram illustrating an example of reference patterns formedon the ground and a vehicle;

FIG. 3 is a flowchart illustrating a process of synthesizing 3D imagesaccording to an embodiment of the present invention;

FIG. 4 is a diagram illustrating images input via a plurality of wideangle cameras;

FIG. 5 is an image illustrating the acquisition of an optical parameterusing an input image;

FIG. 6 is a diagram illustrating Gaussian sphere mapping;

FIG. 7 is a diagram illustrating division for respective longitudes uponmapping Gaussian spheres into a 3D sphere;

FIGS. 8 and 9 are diagrams illustrating the mapping of a ground planeand a non-ground plane upon mapping Gaussian spheres into a 3D sphere;

FIG. 10 is a diagram illustrating a 3D single synthetic image;

FIG. 11 is an image in which the distortion of the adjacent (groundplane) area of FIG. 10 has been corrected; and

FIG. 12 is a diagram illustrating a look-up table.

BEST MODE

Preferred embodiments of the present invention will be described indetail with reference to the accompanying drawings. The same componentsare designated by the same reference numerals throughout theaccompanying drawings as much as possible. Furthermore, detaileddescriptions of the well-known functions and configurations of thepresent invention that may make the gist of the present inventionunnecessarily obscure will be omitted.

FIG. 1 is a diagram illustrating wide angle cameras and an apparatus forsynthesizing 3D images according to an embodiment of the presentinvention, which includes a plurality of wide angle cameras 10, a 3Dimage output device 50, and an apparatus 100 for synthesizing 3D images.

The wide angle cameras 10 include at least four wide angle cameras atany locations in front of, in back of, on the left of, and on the rightof a vehicle. The wide angle cameras 10 photograph reference patterns 1formed on the ground, as illustrated in FIG. 2, and a non-ground plane,convert the photographed images into electrical signals, and thentransmit the electrical signals to the image input/output unit 51 of the3D image output device 50. In this case, if the number of wide anglecameras 10 is, for example, four, it is preferred that the wide anglecameras 10 be arranged such that the coverage areas of neighboring wideangle cameras 10 overlap each other.

The wide angle cameras 10 are based on a concept including not only asimple optical instrument but also an electrical device, such as animage sensor configured to convert optical signals into electricalsignals or the like. For example, if a target object is a vehicle, thewide angle cameras 10 may be arranged in front of, in back of, on theleft and on the right of the vehicle, or on the corners of the vehicle,and the individual cameras 11, 12, 13, 14 may be arranged such that thecoverage areas of neighboring cameras at least partially overlap eachother. The wide angle cameras 10 use fisheye lenses in order to ensureangles of field. The reference patterns 1 are known patterns, and areinstalled on the ground on which fields of view can be ensured by therespective cameras. It is assumed that the physical location informationof the feature points of the reference patterns 1 are known in advance.To ensure the visibility of the reference patterns 1, it is preferableto select as the color of each of the patterns a color having a highdifference in brightness with respect to the color of the background ofthe ground.

The image input/output unit 51 of the 3D image output device receivesphotographed images of the reference patterns 1 formed on the ground anda non-ground plane from the plurality of wide angle cameras 10 mountedon the vehicle, and transmits the plurality of input images to theapparatus 100 for synthesizing 3D images. If necessary, the imageinput/output unit 51 may perform image preprocessing using a filter orthe like. The image input/output unit 51 is part of the 3D image outputdevice 50 mounted on the vehicle, and transmits the images input fromthe plurality of wide angle cameras 10 to the apparatus 100 forsynthesizing 3D images.

The apparatus 100 for synthesizing 3D images stores algorithmsconfigured to correct the plurality of images and synthesize them into3D images. The apparatus 100 for synthesizing 3D images, as illustrated,may include an image arrangement estimation unit 110, a sphere mappingunit 120, a virtual space arrangement unit 130, a single imageacquisition unit 140, a distortion correction unit 150, and an LUTgeneration unit 160.

The image arrangement estimation unit 110 extracts feature points fromthe reference patterns 1 photographed via the plurality of wide anglecameras 10, and estimates the relative location and installation angleof each of the cameras using the known physical location information ofthe extracted feature points. The image arrangement estimation unit 110stores information about the estimated relative location andinstallation angle of the camera. That is, the image arrangementestimation unit 110 estimates only the relative location and inclinedangle at which the camera is disposed based on the reference patterns 1on the ground.

The sphere mapping unit 120 obtains optical parameters including theoptical center of the lens for the correction of lens distortion usingthe images photographed via the wide angle cameras 10, and maps each ofthe images onto the surface of a Gaussian sphere using the obtainedoptical center. In this case, the optical parameters may include one ormore of an optical center, aspect ratio, an image sensor, a projectiontype, and focal length.

The image sensor, the aspect ratio and the focal length may be obtainedthrough the data sheet of the sensors, and the optical center may bedetermined based on the lens circle or center of an ellipse of thephotographed image.

The correction of radial distortion may be viewed as being the same aspin-hole projection in that the refraction of the lens is eliminated. Asprojection equations, four projections equations, that is, anequidistant projection equation, an orthographic projection equation, anequisolid angle projection equation, and a stereographic projectionequation, may be chiefly used.

Once the principal parameters used for the correction of lens distortionhave been obtained as described above, each of the obtained images ismapped onto the Gaussian sphere using the obtained information.

Using the relative location and installation angle of the cameraestimated by the image arrangement estimation unit 110, the virtualspace arrangement unit 130 changes the angle and distance of theGaussian sphere so that the Gaussian sphere coincides with the realreference patterns 1, and then arranges the Gaussian sphere in a 3Dvirtual space. The virtual space arrangement unit 130 may use anoptimization algorithm for estimating change parameters when changingthe angle of rotation and distance of movement of the Gaussian sphere.As the optimization algorithm, an optimization algorithm, such asparticle swarm optimization (PSO), gradient descent (GD), and least meansquare estimation (LMSE) or the like, may be used.

The single image acquisition unit 140 obtains a single 3D image bymapping the individual images arranged in the 3D virtual space onto theinner surface of a 3D sphere corresponding to a single large sphere. Inorder to map an image corresponding to the central portion of each ofthe images, the single image acquisition unit 140 divides the surface ofthe 3D sphere by the number of cameras, and maps only images from themost adjacent cameras. That is, the single image acquisition unit 140divides the inner surface of the 3D sphere for respective longitudesbased on the number of cameras, and divides one surface of the 3D spheredivided for individual longitudes into ground and non-ground planes andmaps the ground and non-ground planes for respective latitudes. In thiscase, a mapping method for a ground plane and a mapping method for anon-ground plane may differ from each other in the 3D sphere.

In the image obtained via the single image acquisition unit 140, thedistortion correction unit 150 corrects the radial distortion of aground plane, that is, the central portion of the image, to arectilinear form.

The LUT generation unit 160 generates a look-up table by associating theindividual pixels of the synthetic image with the individual pixels ofthe input images. That is, the LUT generation unit 160 generates amapping table defining the relationship in which the originalcoordinates of each pixel of the plurality of input images obtained viathe plurality of wide angle cameras 10 have been mapped to the finalcoordinates of each pixel of the synthetic image.

The operation of the apparatus for synthesizing 3D images configured asdescribed above will be described in greater detail with reference tothe accompanying drawings.

FIG. 3 is a flowchart illustrating a process of synthesizing 3D imagesaccording to an embodiment of the present invention.

First, the known reference patterns 1 are installed on the ground, asillustrated in FIG. 2. It is assumed that in the reference patterns 1,the locations of the feature points (the intersections of the patterns)of the individual patterns and the distances between the feature pointsare known to the apparatus 100 for synthesizing 3D images in advance.

The 3D image output device 50 receives a plurality of images obtainedwhen the plurality of wide angle cameras 10 mounted on the vehiclephotographs the reference patterns 1 formed on the ground and transfersthe images to the apparatus 100 for synthesizing 3D images at step S11.

Thereafter, the apparatus 100 for synthesizing 3D images extracts thefeature points of the reference patterns 1 from the images photographed,as shown in FIG. 4, and estimates the relative location and installationangle of each of the cameras using the known physical locationinformation of the extracted feature points at steps S12 and S13. Inthis case, the apparatus 100 for synthesizing 3D images storesinformation about the estimated relative location and installation angleof the camera. That is, the apparatus 100 for synthesizing 3D imagesestimates only the relative location and inclined angle at which thecamera is disposed based on the reference patterns 1 of the ground.

After estimating the relative location and installation angle of thecamera, the apparatus 100 for synthesizing 3D images obtains opticalparameters including the optical center of a lens for the correction oflens distortion using the photographed images and maps each of theimages onto the surface of a Gaussian sphere using the obtained opticalparameters at steps S14 and S15. In this case, the optical parametersmay include one or more of an optical center, aspect ratio, an imagesensor, a projection type and focal length.

The correction of radial distortion may be viewed as being the same aspin-hole projection in that the refraction of a lens is eliminated. Asprojection equations, four projections equations, that is, anequidistant projection equation, an orthographic projection equation, anequisolid angle projection equation, and a stereographic projectionequation, as shown in Equation 1, may be chiefly used. In Equation 1,R_(f) is a projected distance in an image plane, f is a focal length,and φ is the incident angle of an incident ray. Alternative, variousequations may be used.

$\begin{matrix}{{{{Equidistance}\mspace{14mu} {projection}\text{:}\mspace{14mu} R_{f}} = {f \cdot \phi}}{{{Orthographic}\mspace{14mu} {projection}\text{:}\mspace{14mu} R_{f}} = {f \cdot {\sin (\phi)}}}{{{Equisolid}\mspace{14mu} {projection}\text{:}\mspace{14mu} R_{f}} = {2{f \cdot {\sin \left( \frac{\phi}{2} \right)}}}}{{{Stereographic}\mspace{14mu} {projection}\text{:}\mspace{14mu} R_{f}} = {2{f \cdot {\tan \left( \frac{\phi}{2} \right)}}}}} & (1)\end{matrix}$

Meanwhile, the aspect ratio of an image sensor, which is a principalparameter for the correction of camera distortion, can be obtained fromthe data sheet of image sensors. In another method, the aspect ratio maybe obtained in a process of estimating an elliptical shape, that is, theshape of the lens circle of an image photographed as shown in FIG. 5,and converting it into a circular shape.

The optical center may be considered to be the center of the lens circleor ellipse of the corresponding photographed image.

The focal length may be obtained based on values given uponmanufacturing and the data sheet of the image sensors. In anothermethod, the focal length may be defined as a focal length value in thecase where a rectilinear line appears without distortion when thedistortion of an image has been eliminated in accordance with theprojection equations, and thus the focal length may be obtained byestimating the focal length value.

Once the principal parameters used to correct lens distortion have beendetermined as described above, each of the images is mapped onto aGaussian sphere using the obtained information, as illustrated in FIG.6. FIG. 6 illustrates various types of Gaussian sphere mapping methodsbased on projection types. The different types of lines have been usedto distinguish the various projection types.

After mapping each of the images onto the Gaussian sphere as describedabove, the apparatus 100 for synthesizing 3D images changes the angleand distance of the Gaussian sphere so that the Gaussian spherecoincides with the real reference patterns 1, and then arranges theGaussian sphere in a 3D virtual space, using the estimated relativelocation and installation angle of the camera at step 16. In this case,an optimization algorithm for estimating change parameters when changingthe angle of rotation and distance of movement of the Gaussian spheremay be used. As the optimization algorithm, an optimization algorithm,such as particle swarm optimization (PSO), gradient descent (GD), orleast mean square estimation (LMSE), may be used.

The process of performing arrangement in a 3D virtual space as describedabove includes estimating locations and angles at which Gaussian spheresrepresentative of respective cameras are disposed based on the referencepatterns 1 of the ground and then arranging the Gaussian spheres in avirtual space using the estimated values. The Gaussian spheres arearranged in a virtual 3D space, and represent the real cameras mountedon a vehicle. The process of performing estimation is performed byvarying the angle of rotation and the distance of movement in adirection that increases the coincidence with the real referencepatterns 1 when each of the Gaussian spheres is rotated and moved andthen an image mapped onto the Gaussian sphere is mapped onto the ground.Each of the Gaussian spheres may be mapped to the real referencepatterns 1 using the following Equation 2:

XY _(p) =f _(pc)·(S _(c) ·R(ψ)·R(θ)·R(φ))(XY _(c))  (2)

In this equation, XY_(c) is the rectangular coordinates of a point onthe surface of a sphere when the center of a Gaussian sphere is set asan origin and may be represented by (X_(c),Y_(c),Z_(c)), and XY_(p) isthe 2D coordinates of a point at which XY_(c) is mapped onto the ground.S_(c) is a multiplier that is used to increase/reduce the radius of theGaussian sphere, f_(pc) is a function that maps 3D coordinates to apoint in a plane, and R(φ), R(θ) and R(ψ) may be represented by rotationmatrices of 3D coordinates, as expressed in the following Equation 3. Ifthe point of view of the camera is defined as an y axis, a directionorthogonal thereto is defined as an x axis and a direction vertical tothe vehicle is defined as a z axis, φ is an angle at which rotation hasbeen made around the x axis, θ is an angle at which rotation has beenmade around the y axis, and ψ is an angle at which rotation has beenmade around the z axis.

$\begin{matrix}{{{R(\varphi)} = \begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \varphi} & {\sin \; \varphi} \\0 & {{- \sin}\; \varphi} & {\cos \; \varphi}\end{pmatrix}},{{R(\theta)} = \begin{pmatrix}{\cos \; \theta} & 0 & {{- \sin}\; \theta} \\0 & 1 & 0 \\{\sin \; \theta} & 0 & {\cos \; \theta}\end{pmatrix}},{{R(\psi)} = \begin{pmatrix}{\cos \; \varphi} & {\sin \; \varphi} & 0 \\{{- \sin}\; \varphi} & {\cos \; \varphi} & 0 \\0 & 0 & 1\end{pmatrix}}} & (3)\end{matrix}$

When both of Equations 2 and 3 are expanded, the following Equation 4 isobtained. In this case, X_(pd) represents the distance along which theGaussian sphere has moved in the x-axis direction, and Y_(pd) representsthe distance along which the Gaussian sphere has moved in the y-axisdirection.

$\begin{matrix}{\begin{matrix}{X_{p} = {{\pm S_{c}}{\sqrt{X_{c}^{2} + Y_{c}^{2} + Z_{c}^{2}} \cdot}}} \\{= {\left( \frac{\left( {{X_{c}\left( {\cos \; {\theta cos}\; \psi} \right)} + {Y_{c}\left( {\cos \; \theta \; \sin \; \psi} \right)} + {Z_{c}\left( {{- \sin}\; \theta} \right)}} \right.}{\begin{matrix}\left( {{X_{c}\left( {{\sin \; {\varphi sin}\; \psi} + {\cos \; {\varphi sin\theta cos\psi}}} \right)} +} \right. \\{{Y_{c}\left( {{{- \sin}\; {\varphi cos}\; \psi} + {\cos \; {\varphi sin}\; {\theta sin}\; \psi}} \right)} + {Z_{c}\left( {\cos \; {\varphi cos}\; \theta} \right)}}\end{matrix}} \right) + X_{pd}}}\end{matrix}\begin{matrix}{X_{p} = {{\pm S_{c}}{\sqrt{X_{c}^{2} + Y_{c}^{2} + Z_{c}^{2}} \cdot}}} \\{= {\left( \frac{\begin{matrix}\left( {{X_{c}\left( {{{- \cos}\; \varphi \; \sin \; \psi} + {\sin \; {\varphi sin\theta cos}\; \psi}} \right)} +} \right. \\{{Y_{c}\left( {{\cos \; {\varphi cos\psi}} + {\sin \; {\varphi sin}\; {\theta sin}\; \psi}} \right)} + {Z_{c}\left( {\sin \; {\varphi cos}\; \theta} \right)}}\end{matrix}}{\begin{matrix}\left( {{X_{c}\left( {{\sin \; {\varphi sin}\; \psi} + {\cos \; {\varphi sin}\; {\theta cos}\; \psi}} \right)} +} \right. \\{{Y_{c}\left( {{{- \sin}\; {\varphi cos}\; \psi} + {\cos \; {\varphi sin}\; {\theta sin}\; \psi}} \right)} + {Z_{c}\left( {\cos \; {\varphi cos\theta}} \right)}}\end{matrix}} \right) + Y_{pd}}}\end{matrix}} & (4)\end{matrix}$

After pattern matching has been performed using Equations 2 to 4, aparticle swarm optimization (PSO) algorithm may be applied using amethod of changing the angle of rotation and the distance of movement.

After the Gaussian spheres have been arranged in the 3D virtual space,the apparatus 100 for synthesizing 3D images maps the images arranged inthe 3D virtual space onto the inner surface of a 3D sphere singlecorresponding to a large sphere, thereby obtaining a single 3D image atstep S17. When the images are mapped onto the inner surface of the 3Dsphere, the images of the respective Gaussian spheres arranged in the 3Dvirtual space may be mapped onto areas of the inner surface of the 3Dsphere most adjacent to the center of the 3D sphere. The apparatus 100for synthesizing 3D images divides the inner surface of the 3D spherefor individual longitudes based on the number of cameras in order to mapan image corresponding to the central portion of each of the images, asillustrated in FIG. 7. If the number of wide angle cameras 10 is four,the inner surface of the 3D sphere is divided into four areas forrespective longitudes. In FIG. 7, the rectangle inside the 3D sphererepresents a vehicle, and the small circles located in front of, in backof, on the left of and on the right of the rectangle represents thecameras. In this case, alternative long and short dash lines representthe fields of views (FOVs) of the wide angle cameras 10, and the dottedlines represent seam lines indicative of the boundaries between theadjacent cameras. That is, the image of each of the cameras will bemapped to the most adjacent area of the 3D sphere that is divided forindividual longitudes. The reason why the 3D sphere is divided forindividual longitudes as described above is to map only an imagecorresponding to the central portion of the image from each of the wideangle cameras 10 because distortion rate is high in the outer portion ofthe image.

The apparatus 100 for synthesizing 3D images divides the inner surfaceof the 3D sphere divided for individual longitudes into ground andnon-ground planes based on a latitude corresponding to the ground plane,as illustrated in FIG. 8. In FIG. 8, the lower rectangular plate(hatched) represents a ground plane, and the upper rectangular plate(dotted) represents a surface that connects the central portions ofGaussian spheres. Images representative of the non-ground plane aremapped onto the hemisphere above the upper rectangular plate. Theindividual images arranged in the 3D virtual space are divided for theground and non-ground planes, and then mapped onto the correspondinginner surface of the 3D sphere. In this case, methods of mapping imagesonto the ground and non-ground planes of a 3D sphere may differ fromeach other in order to implement a 3D image.

If the overall 3D area is assigned using the mapping method for anon-ground plane, an image of the surroundings of a real vehicle is alsorepresented to be close to a circle, which becomes an obstacle to theintuitive understanding of the surrounds of a user. Accordingly, themore effect visualization of the surroundings of a vehicle can beachieved by mapping images with respect to the adjacent surroundings ofa vehicle in a top view manner. For this reason, mapping for a groundplane assumed to be an adjacent area and mapping for a non-ground planeassumed to be a remote area are performed in different manners.

For example, when the ground plane of the 3D sphere is mapped, asillustrated in FIG. 9, a point (x_(p),y_(p), z_(p)) of the 3D sphere maybe brought from a point of the most adjacent Gaussian sphere and may bemapped onto the 3D sphere, using a method of calculating T of thefollowing Equation 5 and then obtaining x,y,z. In FIG. 9, the rightGaussian sphere is a sphere represented to illustrate the mapping of theground plane, and the left Gaussian sphere is a sphere represented toillustrate the mapping of the non-ground plane.

$\begin{matrix}{{{x_{x} = x_{p}},{z_{s}/z_{p}}}{{y_{s} = y_{p}},{z_{s}/z_{p}}}{x = {x_{c} + {T\left( {x_{s} - x_{c}} \right)}}}{y = {y_{c} + {T\left( {y_{s} - y_{c}} \right)}}}{z = {z_{c} + {T\left( {z_{s} - z_{c}} \right)}}}{{\left( {x - x_{c}} \right)^{2} + \left( {y - y_{c}} \right)^{2} + \left( {z - z_{c}} \right)^{2}} = R^{2}}{T = \frac{R}{\sqrt{\left( {x_{s} - x_{c}} \right)^{2} + \left( {y_{s} - y_{c}} \right)^{2} + \left( {z_{s} - z_{c}} \right)^{2}}}}} & (5)\end{matrix}$

In this case, (x,y,z) are the coordinates of a point on the surface of aGaussian sphere that is mapped to a point (x_(s),y_(s),z_(s)) in aground plane, and (x_(c),y_(c),z_(c)) represents the center of theGaussian sphere. R represents the radius of the Gaussian sphere.

Meanwhile, when the non-ground plane of the 3D sphere is mapped, a point(x_(p),y_(p),z_(p)) of the 3D sphere may be brought from a point of themost adjacent Gaussian sphere and may be mapped onto the 3D sphere,using a method of calculating T of the following Equation 6 and thenobtaining x,y,z.

$\begin{matrix}{{x = {x_{c} + {T\left( {x_{p} - x_{c}} \right)}}}{y = {y_{c} + {T\left( {y_{p} - y_{c}} \right)}}}{z = {z_{c} + {T\left( {z_{p} - z_{c}} \right)}}}{T = \frac{R}{\sqrt{\left( {x_{p} - x_{c}} \right)^{2} + \left( {y_{p} - y_{c}} \right)^{2} + \left( {z_{p} - z_{c}} \right)^{2}}}}} & (6)\end{matrix}$

In this case, (x,y,z) are the coordinates of a point on the surface of aGaussian sphere that is mapped to a point (x_(p),y_(p),z_(p)) in anon-ground plane, (x_(c),y_(c),z_(c)) represents the center of theGaussian sphere, and R represents the radius of the Gaussian sphere.

Through the above-described process, a single 3D image, such as that ofFIG. 10, is obtained.

For the single 3D image generated as described above, the apparatus 100for synthesizing 3D images corrects the radial distortion of a groundplane, that is, the central portion of the image, to a rectilinear form,such as that of FIG. 11, at step S19. In the single 3D image, referencepatterns, that is, a ground plane, are curved, as illustrated in FIG.10, and thus it is necessary to correct the curved reference patterns torectilinear forms.

The correction of distortion is generally intended to eliminate a lensdistortion phenomenon attributable to a difference in curvature byperforming inverse operation on the curvature of a lens in a camerausing a wide angle lens, other than a pinhole camera, and makingincident points uniform around the optical center based on the resultsof the reverse operation. Accordingly, in the process of correctingdistortion, when a model equation (a projection equation) for thecurvature of a lens is given, the degree of distortion is determinedbased on the equation, and only perspective distortion in an imageremains after the correction step.

After correcting the distortion of the 3D image, the apparatus 100 forsynthesizing 3D images generates a look-up table by tracking cameras andoriginal coordinates for the pixels of each image at step S20.

The look-up table is a means for storing image mapping data about therelationship in which each pixel of the plurality of input imagesobtained from the plurality of wide angle cameras 10 has been mapped toeach pixel of the synthetic image, for example, and may be configured inthe form of that shown in FIG. 12.

Referring to FIG. 12, the look-up table may be considered to be a kindof mapping table that defines the relationship in which the originalcoordinates (x,y) of the pixels of the plurality of input imagesobtained from the plurality of wide angle cameras 10 have been mapped tothe final coordinates (t11, t12, . . . , tmn) of the pixels of thesynthetic image. Each of the final coordinates (t11, t12, . . . , tmn)of the synthetic image may be mapped to a plurality of input images. Thereason for this is that when the images obtained from the wide anglecameras 10 are mapped to a planar image, there are cases where eachpixel of the images is mapped in an N:1 correspondence, other than an1:1 correspondence, because the images are distorted images having wideangles of field as described above. For example, the final coordinateT11 may be mapped to three pairs of input original coordinates (x₁,y₂),(x₃,y₅), and (x₄,y₆). A number of look-up tables equal to the number ofinput images obtained from the wide angle cameras 10 are provided, andcorresponding coordinate values of the synthetic image are included foreach of the input images.

In the process of generating such a look-up table, when the inverseoperation of a required operation is performed in order to obtain outputfor each pixel of a sample synthetic image to generate the look-uptable, the coordinates of each pixel of the input images correspondingto each pixel of the synthetic image can be obtained.

In the process of generating such a look-up table, for example, any onepixel is selected from among pixels that constitute the 3D syntheticimage. The pixel is selected based on its coordinates, and thecoordinates become a final coordinate of the synthetic image.

Once the pixel has been selected, one of the plurality of wide anglecameras 10 that generated the selected pixel is determined. This may beviewed as the reverse process of the synthesis process. In order todetermine one of the plurality of wide angle cameras 10 that generatedthe selected pixel, it may be convenient to use a method of addingidentifiers capable of identifying the plurality of wide angle cameras10 to the input images generated by the cameras 10 and then checking theidentifiers later.

Once the above process has been performed, the original coordinates ofthe final coordinates of a selected pixel in the input image obtained bythe camera may be determined. Accordingly, original coordinatescorresponding to the final coordinates of a specific pixel are obtainedand recorded.

Original coordinates corresponding to the final coordinates of any pixelmay be obtained by sequentially performing this process on all thepixels of the synthetic image.

A look-up table (LUT), such as that of FIG. 12, may be generated bymapping original coordinates obtained using the above-described methodto the final coordinates of corresponding pixels. In this case, all thepixels of the input images are not mapped to the final coordinates ofthe synthetic image. This means that unnecessary pixels of the inputimages may not be mapped to final coordinates and may be discarded.Generally, since only pixels in a specific area of each of the inputimages, that is, only 20% to 50% pixels, are converted into thesynthetic image, the mapping process is performed only on the pixels ofthe input images converted into the synthetic image while referring tothe look-up tables, and thus load and time can be reduced upon imageprocessing.

After generating look-up tables, such as that that of FIG. 12, theapparatus 100 for synthesizing 3D images transmits the generated look-uptables to the 3D image output device 50 mounted on the vehicle, therebyinstalling the look-up tables in the storage unit 55. Thereafter, thevehicle directly maps pixels of images, such as that of FIG. 4, inputvia the plurality of wide angle cameras 10 upon real parking or driving,onto the final synthetic image while referring to the look-up tablesstored in storage unit 55 without performing the synthesis process ofthe present invention, thereby simply and rapidly generating anddisplaying a 3D surrounding image, such as that of FIG. 11.

Accordingly, in the present invention, it is possible to convert a pointof view using only three-axis rotation for a 3D object based on 3Dinformation that is generated while a plurality of cameras is beingprojected onto a single virtual camera, without using a homographicviewpoint conversion algorithm, which is a conventional spatialtransformation method. The present invention is characterized in thatboth ground plane and non-ground plane information can be synthesized byperforming image synthesis based on the 3D modeling of a space around avehicle.

The apparatus and method for synthesizing 3D images are not limited tothe configuration and operation of the above-described embodiments. Theabove-described embodiments may be configured such that all or some ofthe embodiments are selectively combined, thereby making variousmodifications.

1. An apparatus for synthesizing three-dimensional (3D) images,comprising: an image input/output unit configured to receivephotographed images of a reference pattern formed on a ground and anon-ground plane from a plurality of wide angle cameras mounted on avehicle, and to output the photographed images; an image arrangementestimation unit configured to extract feature points from the referencepatterns of the image input from the image input/output unit, and toestimate a relative location and installation angle of each of thecameras using known physical location information of the extractedfeature points; a sphere mapping unit configured to obtain opticalparameters for correction of lens distortion including an optical centerof a lens using the photographed images, and to map each of the imagesto a surface of a Gaussian sphere using obtained optical center; avirtual space arrangement unit configured to, using the relativelocation and installation angle of the camera estimated by the imagearrangement estimation unit, change an angle and distance of theGaussian sphere so that the Gaussian sphere coincides with the realreference patterns, and arrange the Gaussian spheres in a 3D virtualspace; and a single image acquisition unit configured to obtain a single3D image by mapping the images arranged in the 3D virtual space onto aninner surface of a 3D sphere corresponding to a single large sphere. 2.The apparatus of claim 1, wherein the image arrangement estimation unitsets information about the estimated relative location and installationangle of the camera.
 3. The apparatus of claim 1, wherein the opticalparameters comprise one or more of an optical center, aspect ratio, animage sensor, a projection type and focal length.
 4. The apparatus ofclaim 1, wherein the virtual space arrangement unit uses an optimizationalgorithm for estimating change parameters when changing an angle ofrotation and distance of movement of the Gaussian sphere.
 5. Theapparatus of claim 4, wherein the optimization algorithm is any one ofparticle swarm optimization (PSO), gradient descent (GD), and least meansquare estimation (LMSE).
 6. The apparatus of claim 1, wherein thesingle image acquisition unit, in order to map an image corresponding toa central portion of each of the images, divides the surface of the 3Dsphere by a number of cameras, and maps only an image from a mostadjacent camera.
 7. The apparatus of claim 1, wherein the single imageacquisition unit divides the inner surface of the 3D sphere forindividual longitudes based on a number of cameras, divides one surfaceof the 3D sphere divided for the individual longitudes into ground andnon-ground planes, and performs mapping for individual latitudes.
 8. Theapparatus of claim 7, wherein mapping methods for ground and non-groundplanes differ from each other in the 3D sphere.
 9. A method ofsynthesizing 3D images, comprising: receiving, by a plurality of wideangle cameras mounted on a vehicle, a plurality of images of referencepatterns formed on a ground; extracting feature points from thephotographed reference patterns, and estimating a relative location andinstallation angle of each of the cameras using known physical locationinformation of the extracted feature points; obtaining opticalparameters for correction of lens distortion including an optical centerof a lens using the photographed images, and mapping each of the imagesonto a surface of a Gaussian sphere using the obtained optical center;using the estimated relative location and installation angle of thecamera, changing an angle and distance of the Gaussian sphere so thatthe Gaussian sphere coincides with the real reference patterns, andarranging the Gaussian spheres in a 3D virtual space; and obtaining asingle 3D image by mapping the images arranged in the 3D virtual spaceonto an inner surface of a 3D sphere corresponding to a single largesphere.
 10. The method of claim 9, wherein arranging the Gaussianspheres in the 3D virtual space is performed by using any one of PSO,GD, and LMSE algorithms in order to estimate change parameters whenchanging the angle of rotation and distance of movement of the Gaussiansphere.
 11. The method of claim 9, wherein arranging the Gaussianspheres in the 3D virtual space comprises performing matching with thereal reference patterns using the following equation:XY_(p) = f_(pc) ⋅ (S_(c) ⋅ R(ψ) ⋅ R(θ) ⋅ R(φ))(XY_(c)) where${{R(\varphi)} = \begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \varphi} & {\sin \; \varphi} \\0 & {{- \sin}\; \varphi} & {\cos \; \varphi}\end{pmatrix}},{{R(\theta)} = \begin{pmatrix}{\cos \; \theta} & 0 & {{- \sin}\; \theta} \\0 & 1 & 0 \\{\sin \; \theta} & 0 & {\cos \; \theta}\end{pmatrix}},{{R(\psi)} = \begin{pmatrix}{\cos \; \varphi} & {\sin \; \varphi} & 0 \\{{- \sin}\; \varphi} & {\cos \; \varphi} & 0 \\0 & 0 & 1\end{pmatrix}},$ where XY_(p) is a point on the inner surface of thesphere, XY_(c) is rectangular coordinates of a point on the innersurface of the sphere when a center of the Gaussian sphere is set as anorigin, S_(c) is a multiplier that is used to increase/reduce a radiusof the Gaussian sphere, f_(pc) is a function that maps 3D coordinates toa point in a plane, R(φ), R(θ) and R(ψ) are rotation matrices of 3Dcoordinates, φ is an angle at which rotation has been made around an xaxis, θ is an angle at which rotation has been made around an y axis,and ψ is an angle at which rotation has been made around an z axis. 12.The method of claim 9, wherein obtaining the single 3D image comprisesmapping the images arranged in the 3D virtual space onto areas of theinner surface of the 3D sphere most adjacent to a center of the 3Dsphere.
 13. The method of claim 9, wherein obtaining the single 3D imagecomprises: dividing the inner surface of the 3D sphere for individuallongitudes based on a number of cameras in order to map imagescorresponding to central portions of the images; dividing the innersurface of the 3D sphere divided for the individual longitudes intoground and non-ground planes based on a latitude corresponding to theground plane; and dividing the images arranged in the 3D virtual spaceinto ground and non-ground planes, and performing mapping ontocorresponding areas of the inner surface of the 3D sphere.
 14. Themethod of claim 13, wherein performing the mapping is performed by usingdifferent mapping methods for the ground and non-ground planes.
 15. Themethod of claim 13, wherein performing the mapping comprises, whenperforming mapping onto the ground plane of the 3D sphere, bringing apoint (x_(p),y_(p), z_(p)) of the 3D sphere from a point of a mostadjacent Gaussian sphere and mapping the point (x_(p),y_(p),z_(p)) ofthe 3D sphere onto the 3D sphere, using a method of calculating T of thefollowing equation and then obtaining x,y,z: x_(x) = x_(p), z_(s)/z_(p)y_(s) = y_(p), z_(s)/z_(p) x = x_(c) + T(x_(s) − x_(c))y = y_(c) + T(y_(s) − y_(c)) z = z_(c) + T(z_(s) − z_(c))(x − x_(c))² + (y − y_(c))² + (z − z_(c))² = R²$T = \frac{R}{\sqrt{\left( {x_{s} - x_{c}} \right)^{2} + \left( {y_{s} - y_{c}} \right)^{2} + \left( {z_{s} - z_{c}} \right)^{2}}}$where (x,y,z) are coordinates of a point on the surface of the Gaussiansphere that is mapped to a point (x_(s),y_(s),z_(s)) in the groundplane, (x_(c),y_(c),z_(c)) represents the center of the Gaussian sphere,and R represents a radius of the Gaussian sphere.
 16. The method ofclaim 13, wherein performing the mapping comprises, when performingmapping onto the ground plane of the 3D sphere, bringing a point(x_(p),y_(p),z_(p)) of the 3D sphere from a point of a most adjacentGaussian sphere and mapping the point (x_(p),y_(p),z_(p)) of the 3Dsphere onto the 3D sphere, using a method of calculating T of thefollowing equation and then obtaining x,y,z:x = x_(c) + T(x_(p) − x_(c)) y = y_(c) + T(y_(p) − y_(c))z = z_(c) + T(z_(p) − z_(c))$T = \frac{R}{\sqrt{\left( {x_{p} - x_{c}} \right)^{2} + \left( {y_{p} - y_{c}} \right)^{2} + \left( {z_{p} - z_{c}} \right)^{2}}}$where (x,y,z) are coordinates of a point on the surface of the Gaussiansphere that is mapped to a point (x_(s),y_(s),z_(s)) in the groundplane, (x_(c),y_(c),z_(c)) represents the center of the Gaussian sphere,and R represents a radius of the Gaussian sphere.